MobileWorks: Designing for Quality in a Managed Crowdsourcing Architecture (Extended Abstract)

Online labor marketplaces offer the potential to automate a variety of tasks too difficult for computers, but present requesters with significant difficulties in obtaining accurate results. We share experiences from building MobileWorks, a crowd platform that departs from the marketplace model to provide robust, high-quality results. Three architectural contributions yield measurably improved accuracy on input tasks. A dynamic work routing system identifies expertise in the crowd and ensures that all work posted into the system is completed with bounded completion times and at fair worker prices. A peer management system ensures that incorrect answers are prevented by experienced members of the crowd. Last, social interaction techniques give the best workers the ability and incentives to manage, teach & supervise other members of the crowd, as well as to clarify tasks. This process filters worker error and allows the crowd to collectively learn how to solve unfamiliar tasks.